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eam eibbdadead eat a Gra ec Ven 


9 November 1983 


ob SA 
NOTE TO: Director of Central Intelligence” 


FROM : Deputy Director for Intelligence 


SUBJECT: Hughes Study on Collection Systems 


Attached is the study we have been discussing. 
I have outlined some key points in the first dozen 
pages and call your attention to further highlighting 
on pages 17, 33, 59, 71, 78, and 92. The summary 
is useless. 


Out West 
One point brought out in my pricring/that I 


did not find in the report is that in assessing 
alternative collection systems you would select 

a half dozen or so samples like the surface to 
air missile sample discussed in the study to give 
you a feel for comparative advantages of systems 
in diverse areas. For example, you might want to 
look at a particular economic target, two or three 
other kinds of military targets and perhaps a 
politicalsmilitary target to give you a relative 
sense of the advantages and disadvantages of each 
of the candid collection systems. You obviously 
cannot apply this cumbersome methodology to all 
possible targets. Samplings would be required. 


I should caution again that Hughes acknowledges 
more work remains to be done on this, amd that it is 
cumbersome and time consuming, and my impression is 
that they are continuing this work. 


IG. 
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FOREWORD 


This interim report was prepared by the Hughes Aircraft Company, 
Fullerton, California under contract 82-N7880000. The prini¢ipal ‘Hughes: 
analysts for this effort have been Ms. K.J. Baldasari, Mr. R.S. McKinnell, 
and Mr. W. Lund. : EERO Ta 
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I. PROBLEM STATEMENT 


PROBLEM STATEMENT 


DEVELOP A METHODOLOGY TO PERFORM UTILITY ASSESSMENTS OF 
COLLECTION CONCEPTS 


— THE METHODOLOGY SHOULD BE CAPABLE OF DEALING WITH 
THE ENTIRE CONCEPT LIFE CYCLE 


RELATE THE METHODOLOGY TO SPECIFIC INTELLIGENCE QUESTIONS 


The purpose of this effort is to develop a methodology to assess the 
utility and capabilities of collection system concepts. Such a methodology is 
needed to bring a systematic procedure to a process which has heretofore been’ 
intuitive and variable. 

The methodology developed must be flexible, enabling it to be applied to 
the wide range of concerns and problems with which collection systems must .- 
deal, The final task of this project therefore is to show the developed - 
methodology's flexibility and adaptability by applying the outlined proeedures 
to a mutually agreed upon problem, assessing the information required to 
determine the impact of a new surface-to-air missile (SAM) on the balance of 
power. 
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pase secs esseeceecs 


METHODOLOGY OBJECTIVES 


Risk Ge Sed we ola eter a! 


e DEVELOP A STANDARDIZED PROCEDURE FOR ASSESSING COLLECTION 
TECHNIQUES AND STRATEGIES 


e STRUCTURALLY INDEPENDENT OF PROBLEM TYPE 
e HANDLE MULTI-OBJECTIVE. MULTI-FACTOR DECISIONS 


¢ GENERATE UNIQUE, TRACEABLE PATHS FROM PROBLEM ELEMENTS 
THROUGH INFORMATION REQUIREMENTS TO APPLICABLE COLLECTION 
TECHNIQUES 


e METHODOLOGY SHOULD DEVELOP A MEANS TO FOCUS ON 
SIGNIFICANT ISSUES, SCAN THELESS SIGNIFICANT, ELIMINATE THE 
IRRELEVANT 


e INCORPORATE MULTI-FACETED TYPES OF DATA AND INFORMATION 
© COMBINE QUALITATIVE AND QUANTITATIVE MEASURES OF 
‘EFFECTIVENESS 
e INCORPORATE UNCERTAINTY IN PARAMETER QUANTIFICATION 
e ALLOW FOR EASY INCORPORATON OF ADDITIONAL INFORMATION 
e AGGREGATE EXPERT OPINIONS 


360878 61627 82) 


The first step in methodology delineation involved documenting 
objectives the methods employed should satisfy. The overriding objective was 
to develop an approach which would replace the intuitive methods employed in 
the past with techniques which are auditable, generate traceable analysis” 
paths and are valid over time. - 

The need for a standardized procedure is especially acute given the 
diverse nature of problems to be addressed and the multi-faceted decisions to 
be reached. The methodology should highlight the critical decision components 
in an easy manner and allow for analysis review and sensitivity testing. ; 

A complicating factor in the problem is the diverse nature of the data 
used in collection analysis. The data tends to range from being highly —~ 
qualitative, such a the doctrine of the enemy, to the highly quantitative, 
such as the physical number of systems to be deployed. Additionally, the 
collectibility of required information is not a guaranteed event, but instead 
is uncertain and possesses a probability distribution whose form may depend 
upon the expert performing the analysis or formulating the question. 
Furthermore, the data employed does not remain constant, but instead is 
dynamic and depends upon the timing of the analysis and its requirement. 
Therefore, to reflect this diversity: the methodology should be adaptable to 
reapplication and easily incorporate additional information as it becomes 
available. 
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II. METHODOLOGY DEVELOPMENT 


DECISION ANALYSIS 


DECISION ANALYSIS COUPLES THE QUANTITATIVE 
TECHNIQUES OF SYSTEMS ANALYSIS WITH THE 
QUALITATIVE THEORIES OF DECISION THEORY 


DECISION ANALYSIS !S SUITED TO PROBLEMS 
CHARACTERIZED BY 


— UNCERTAINTY _ 

— COMPLEX PREFERENCES 
— IMPORTANCE 

— UNIQUENESS 

— LONG RUN IMPLICATIONS 


3606 78-6 (6.2783) 


Decision analysis refers to a body of knowledge and techniques which 
combines decision theory and system analysis. Decision theory investigates 
the manner in which people assemble information to make decisions and the 
actual decision making process. System analysis brings to the effort 
techniques for problem decomposition, handling uncertainty in parameters, and 
mathematical decision rules. 

The combination of these two disciplines is ideally suited to the 
collection systems problem since it allows for uncertainty in parameter 
evaluation, can handle complex problems through decomposition, and provides a 
systematic framework for evaluation, Decision analysis is not appropriate fdr 
smaller problems as the effort of analysis may be too large for less 
significant issues. Therefore, the techniques are most appropriate when the 
issues under study are complex, of relative importance, unique and . 
identifiable, and the long-run implications are such that a temporarilly 
stable method of review is desirable. 


- 
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SUCCESSFUL DECISION ANALYSIS 
APPLICATIONS 


VOYAGER SPACE PROJECT 


NUCLEAR POWER PLANT SITE SELECTION 


URBAN DEVELOPMENT 


FEDERAL ENERGY POLICY 
CORPORATE INVESTMENT POLICY 


DAM SITE SELECTION 


360479 916278); 


Decision analysis is a proven, developed approach through which 
decisions have been made. Decision analysis has been applied on projects 
ranging from the highly quantitative Voyager Space Project to the qualitative 
concerns of urban development. The broad range of successful applications 
illustrates the flexibility of the method as well as its maturity. 

Selection decisions have been made successfully using these techniques. 
By its very nature, decision analysis is well suited to comparing alternatives 
and making incremental assessments. 7 


=! G5 = 
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DECISION ANALYSIS: THE PRO’S 


e THE DECISION MAKER IS ENCOURAGED TO SCRUNTINIZE THE PROBLEM 
AS AN ORGANIC WHOLE 


e ALLOWS EACH EXPERT TO GIVE TESTIMONY ABOUT A PARTICULAR AREA 
OF EXPERTISE IN AN UNAMBIGUOUS, QUANTITATIVE MANNER THAT 
CAN BE INTEGRATED INTO THE OVERALL ANALYSIS 


e ALLOWS THE DISTINGUISHING OF THE DECISION MAKER’S 
PREFERENCES FOR CONSEQUENCES FROM JUDGEMENTS ABOUT 
UNCERTAINTIES 


e CAN BE USED TO CLEARLY COMMUNICATE THE RATIONALE FOR THE 
ADOPTED STRATEGY AND RALLY SUPPORT FOR IT 


e CLEARLY POINTS OUT THE FACTORS INCORPORATED INTO THE 
ANALYSIS 


e ALLOWS FOR THE DECOMPOSITION OF THE PROBLEM INTO ITS BASIC 
PARTS SO ISSUES WHERE THERE ARE FUNDAMENTAL DISAGREEMENTS 
CAN BE EXPOSED 


e MAY REVEAL PREVIOUSLY HIDDEN RELATIONSHIPS/STRUCTURE 


360479 7 1627-03) 


Decision analysis brings several advantages to a collection system 
analysis. The techniques encourage separation of the factors and basis of the 
decision from the personal biases and feelings of the decision maker. 
Additionally, structure is introduced to the problem which allows for easier 
communication of ideas and improved analysis possibilities. The framework is 
such that sensitivity analysis is possible. Under the intuitive methods of 
the past such analysis of variance was not possible. 

Decision analysis makes complex problems easier because decomposition 
into smaller problem pieces is possible. Decomposition allows those most 
capable of performing a subtask to do so without also requiring performance 
outside any areas of expertise. Synergy will result when the pooled opinions 
of these experts are generated which will lead to a better basis for reaching 
decisions than otherwise would have been available. 
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DECISION ANALYsIS DOES NOT REPLA 


CE OR LESSEN THE VALUE 
OF THE EXPERT'S OPINION 


THE DEVELOPED METHODOLOGY DO 
PROBLEM ELEMENTS ARE CONSIDERED 


ES NOT ENSURE ALL 


While decision anal 
without its weaknesses, 
recognized going into at 
first limitation of decis 


ysis offers many advantages, the methodology is not 
As with any technique, these limitations need to be 
ask so their potential effects can be minimized. The 
still require input from 

The methodology will not 
e or experience, but the method will take 
d make optimal use of the information, © 
automatically considered. 
Structured from the ground 
are, however 
discovered, 
work on the p 
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METHODOLOGY CHOSEN — 
DECISION ANALYSIS 


OBJECTIVE DECISION ANALYSIS SOLUTION 


e STANDARDIZED e TREE STRUCTURE SUITED TO WIDE VARIETY OF 
PROCEDURE PROBLEMS WITH MULTIPLE OBJECTIVES 


e GENERATE TRACEABLE e TREE ANALYSIS HIGHLIGHTS CRITICAL PATHS 
PATHS AND PARAMETERS OF PROBLEM 


e INCORPORATE MULTI- e TREE STRUCTURE HANDLES QUALITATIVE AND 
FACETED DATA AND QUANTITATIVE DATA 


INFORMATION UNCERTAINTY IS EASILY HANDLED BY 
SENSITIVITY ANALYSIS ! 


@ EASILY UPDATED WITH NEW INFORMATION 


260473-8 (627 63; 


To verify decision analysis is a desirable methodology, a comparison is 
needed between the. outlined objectives and the advantages of decision 
analysis. From the above chart, it is apparent decision analysis satisfies 
each of the outlined criteria and is therefore suited to the task outlined. 

The remainder of this report addresses the specifics of applying the 
methodology to collection system problems. To do this a simple problem of 
information collection and evaluation will be used for illustrative purposes. 
By working a simple problem in detail, it is felt a more in-depth delineation 
of the methodology is obtained and problems which might be expected are-< 
highlighted and resolved. 

The result of using this methodology is the introduction of structure 
and a standardized procedure to what has previously been an intuitive, 
unstructured analysis, The methodology will not replace or reduce the 
importance of expert opinion. The expert's opinion will, however, be more’ 
visible and available for review and discussion, 


ee oe 
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DETERMINATION OF TREE STRUCTURE AND THE 
ASSIGNMENT OF BRANCH WEIGHTS 


DEFINITION OF COLLECTION CONCEPT AND THE 
DETERMINATION OF COLLECTION UNCERTAINTIES 


APPLICATION OF OTHER FACTORS AND SENSITIVITY 
ANALYSIS 


e TIMELINESS 
e SURVIVABILITY 


360579-9 (627.83; 


The basic approach of the defined methodology divides into three 
phases. The first phase involves formulating the problem to be addressed in 
_ an information tree structure. This “tree” depicts the relationships and’ 
hierarchy of the problem from the top of the tree, the question to be © 
answered, to the bottom of the tree, the collectable data items which combine 
to answer the overall issue. The intermediate stages of the tree form the 
linkages between collectable data and the overall question, Information at 
these levels might include system parameters, doctrine considerations, .and 
deployment questions. : 

Once the structure has been outlined, weights are assigned to the 
branches to indicate a particular branch's importance in answering the 
overriding issue at the next highest level or node. A value between O and 1 
is used to define this importance, 0 indicating no relevance (the branch could 
be eliminated from the tree) and 1 indicating this branch is the only relevant 
branch (all others emanating from this node could be eliminated). 
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With the tree structure complete and the weights assigned, the second 
phase begins with specification of the collection concepts along with their 
capabilities. Defining the concept involves establishing the probabilities 
the concept can collect the data described at the lowest levels of the tree. 
The probability assessment will in practice take the form of a distribution 
describing the likelihood of successful collection, By combining the concept 
definition with the tree structure, a result is obtained which delineates the 
value of information available to answer the overriding question of the tree. 

Incorporation of outside factors not included in the tree structure or 
concept definition is accomplished in phase three. Considerations such as the 
availability or timeliness of a concept, or the availability of data from_a 
given concept versus another, can be included in the analysis through the use 
of a discount factor. By allowing for the time value of information in this 
manner, concepts whose timing of availability differ can be compared. 
Additionally, concepts whose expected lifetimes differ can be analyzed in a 
Similar manner. The additional information available because one system will 
have a longer lifetime than another can be handled by “compounding” the value 
obtained by the tree and concept definition analysis. The result achieved by 
completing phase three will be a Probability distribution showing the 
information availability and its likelihood of collection. Using these 
distributions, alternative concepts can be evaluated keeping in mind risk 
aversion, minimum requirements or other factors. 


- ll ~ 
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III. ILLUSTRATIVE EXAMPLE OF METHODOLOGY 


EXAMPLE PROBLEM 


= 
a 
a 


JOHN IS CONSIDERING BUYING A HOUSE. HE HAS 
DRIVEN PAST ONE ON THE WAY TO WORK THAT LOOKS 
LIKE A GOOD POSSIBILITY 

. HE NOW HAS TO FIND INFORMATION ON HOW WELLTHIS 
HOUSE FITS HIS NEEDS — BOTH OBJECTIVE AND 
SUBJECTIVE 
HE WANTS TO KNOW 

e@ WHAT IS THE BEST WAY TO COLLECT THE 
INFORMATION HE NEEDS TO MAKE A DECISION 

STEP NO. 1: BUILD AN INFORMATION VALUE TREE 


STEP NO. 2: ASSESS THE VALUE OF THE INFORMATION 
THAT CAN BE GATHERED BY COMPETING INFORMATION 
COLLECTION CONCEPTS 


STEP NO. 3: EVALUATE OTHER FACTORS IMPACTING 
COLLECTION PERFORMANCE 


360678 11 1627 8) 


What follows is an in-depth discussion of the methodology which will 
describe why it can be such a powerful decision support tool and, more 
specifically, 

- the techniques necessary to apply it, 

- the information it can provide, 

i the insight it will give to both analysts and decision makers. 

As a supplement to this technical discussion, a simple example will be used to 
illustrate the key concepts of the methodology. Please note, however, 

- the example is by necessity somewhat over-simplified in order to 
demonstrate the concepts; it is not intended to be completely 
realistic, and 

- so that it can provide insight into how such an analysis is done, 
the example will be presented in a very detailed and numerically 
intensive manner, 


on 
ae 
q penne 
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The example centers around John who is considering buying a house. He 
has driven by one on the way to work that looks like a good possibility and 
now must collect information on how well this particular house fits his 
needs. Specifically he wants to know: : 

“What is the single most effective method to use to collect the 

information necessary to make this decision.” 


The basic steps John will need to follow to perform his analysis are: 

(1) Build an information value tree to delineate the raw data that 
should be collected. as well as the relative value of each piece to 
the main question. 

(2) Assess collection concept performance by evaluating the ability of 
each candidate collection concept to meet each raw data requirement. 

(3) Incorporate any other relevant factor and perform sensitivity 
analyses. 


Note: This example has been specifically designed to illustrate all facets of 
the methodology and the reader should not be concerned that the techniques 
seem laborious and involved. In actual analyses many of the steps illustrated 
will not be necessary because sensitivities of the problem will allow some 


simplification. 
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III. A. INFORMATION TREE DEVELOPMENT 


INFORMATION VALUE TREE 


REPRESENTS THE RELATIONSHIP BETWEEN DIFFERENT LEVELS OF 
INFORMATION NEED AND THE DATA ELEMENTS THAT SUPPORT THEM 


DEVELOPED BY DRIVING OOWN ONE LEVEL OF INFORMATION AT A TIME 
FROM THE DECISION MAKER‘S QUESTION TO THE AVAILABLE DATA 


DEPICTED GRAPHICALLY BY 
e NODES — REPRESENTING SOME QUANTUM OF INFORMATION 


e BRANCHES — REPRESENTING EITHER 
— THE TIE UP TO THE NEXT HIGHER LEVEL OF INFORMATION 
REQUIRING THIS PIECE OF INFORMATION 


— THE TIES DOWN TO THE LOWER LEVELS OF INFORMATION 
ELEMENTS SUPPORTING THIS PIECE OF INFORMATION 


ESTABLISHES THE RELATIVE VALUE BETWEEN ALL ELEMENTS OF 
INFORMATION RELATIVE TO EITHER 


e A PARTICULAR NODE 
e A PARTICULAR INFORMATION LEVEL 


36047612 (627-83) 


An information value tree is a construct for relating different levels 
of information requirements and the raw data supporting them. It is a 
systematic structuring of how these information and data items relate. The 
information value tree is essentially a graph with 


branches - which represent either data or pieces of information 
aggregated from lower levels of information or raw data, and 
nodes as which represent the tieing together of the data or 


aggregated information to support a specific higher Level 

information need, that is, a further aggregation of 

information. 
The purpose of such a structure is to establish the value of each information 
need or data item relative to the particular information requirement (node) it 
supports as well as to any other information need or data item within the tree. 

In structuring the information tree, the nodes employed can either be 

“or” or “and” in character. The node type selected will depend upon the 
structure and information required by the developer. Whichever structure type 
provides the greatest visibility to the developer should be employed. The 
only precaution needed for the actual analysis is the numerical combinations 
performed at each node must be consistent with the node type. For “and" nodes 
all branches into a node should be- summed yielding the nodal result and for 


“or" nodes only the branches which are active, actually employed or satisfied, 
should be summed. 
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At the top of the next page is the tree John developed to evaluate which 
collection concept provides the most effective support relative to his main 
question: should he consider buying this house. This question is called the 
“decision maker question.” From the tree it can be seen that for John, the 
utility or value of a house is solely a function of 

~- whether it fulfills certain critical needs he has established, 

<= the extent to which the house and yard will need to be cared for, and 

- the investment required and the return expected. 

These are the “critical issues” that drive the decision maker's question. In 
turn, each critical issue is defined by a set of “problem characteristics." 
For example, the investment potential has been broken down into the immediate 
affordability, the continuing affordability, and the future worth. The 
problem characteristics are themselves derived from “data items" which relate 
to pieces of information that are actually collected. Note that the complete 
set of data items that support the decision maker's question are on the far 
right side and range from the number of bedrooms in the house to the planned 
developments in the local area. 

In general, the key concepts to consider at each level of the tree are 
as follows: 

Decision Maker Question ~- the main question against which the collection 

concepts are being evaluated. 

Critical Issues - the key issues or subquestions that drive the resolu- 

tion of the decision maker question. These are still from a 
qualitative vs a quantitative perspective. 

System/Problem Characteristics - quantifiable information that defines 

the key drivers behind the critical issues. 

Data Items - data that is directly collected or directly derived from 

collected data, 

These levels are guidelines to help the analyst in systematically 
constructing the tree structure such that all the relevant issues are 
addressed and the key data requirements are delineated. The structuring 
process can be done solely by the decision maker, as in this example, or in 
concert with a group of experts in the various areas spanned by the tree and 
is somewhat of an art. It is critical to be as systematic as possible; all 
the relevant information needs to be captured and, conversely, as little as 
possible of the extraneous. A key sign that some part of the tree's structure 
is essentially complete occurs when review of that section produces 
discussions revolving around semantics which generate no new insight or data . 

The information represented at different levels of the tree ranges from 
very qualitative at the decision maker question level (e.g., should John < 
consider buying this house) to very quantitative (e.g., the number of bedrooms 
in the house). Specifically, quantitative information can be described using 
numbers; examples might be the number available, dollars spent, percent lost, 
distance traveled, probability of occurrence. Qualitative information, on the 
other hand, is sometimes referred to as “intangible”; examples might be margin 
for safety, ability to predict, impact on the status quo, level of 
performance. One of the strengths of this decision analysis approach is the 
ability to tie together such different types of data. 
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JOHN’S TREE 


8IG ENOUGH 
FUNCTIONAL 
NEEDS 
FULFILLED 


LOCATED WELL 
RELATIVE TO 


MAINTAINABILITY 


SHOULD JOHN BUY} SUPPORT 


HOUSE? REQUIRED 
ae OURABILITY 
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NUMBER OF BEDROOMS 
TOTAL SQUARE FOOTAGE 
SCHOOLS 

WORK 

SHOPPING 

AMOUNT OF YARD WORK 
AGE OF HOUSE 


CONDITION OF SIMILAR HOUSES 
DISASTER VULNERABILITY 


(FIRE. EARTHQUAKE. ETC) 
DOWN PAYMENT 
CLOSING COSTS 


IMMEDIATE 
AFFORDABILITY 


BASIC MONTHLY PAYMENT 
N R ARIANA 
CONTINUING PROJECTED 


AFFORDABILITY 


INVESTMENT 


rots POSSIBLE ASSESSMENTS 


EXPECTED APPRECIATION 


OLE AicliLs PLANNED DEVELOPMENTS 


: DECISION MAKER CRITICAL PROBLEM 
360479-22 (6-27-83) QUESTION CHARACTERISTICS DATA {TEMS 


LEVELS OF THE TREE STRUCTURE 


DECISION MAKER QUESTION 
@ ULTIMATE QUESTION OR PROBLEM TO BE 
ADDRESSED BY REMAINDER OF THE TREE 


CRITICAL ISSUES (KEY SUBQUESTIONS) 


@ QUALITATIVE QUESTIONS WHOSE SOLUTIONS ARE 
NECESSARY TO ANSWER DECISION MAKER 
QUESTION 


SYSTEM/PROBLEM CHARACTERISTICS 

@ QUANTITATIVE SYSTEM PARAMETERS WHICH 
COMBINE TO PROVIDE AN ANSWER TO CRITICAL 
ISSUES 

RAW AND DERIVED DATA 


@ QUANTITATIVE DATA WHICH IS COLLECTED IN ITS 
RAW FORM BY COLLECTION SOURCE OR EASILY 
DERIVED FROM SUCH DATA 


QUALITATIVE 
PERSPECTIVE 


QUANTITATIVE 
PERSPECTIVE 


2604 70-00 (6-27-43) 
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of branches; this is not a requirement, however. More complex questions where 
more detail is necessary or even just desired may have several tiers of 
- branches and nodes at each level. Additionally, the number of tiers or 

branches at a particular level does not have to be consistent throughout the 
tree. For example, instead of just stopping with a branch corresponding to 
the relative distance of the house to work, John could have put a node at the 
end of that branch and added two new branches that corresponded to the 
distance relative to where he works and relative to where his wife works. The 
more detailed the branch structure becomes, the more information it captures 
-and the more insight it provides into the data and issues that drive the 
decision maker's question. Care must be taken, however, to avoid excessive 
detail; for example, the specific distance to every single school in the local 
area is not of any concern to John, so to break the tree down further in this 
area is undesirable. 
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BRANCHING CRITERIA 


' COMPLETENESS 


@ THE SET OF BRANCHES SHOULD CHARACTERIZE ALL 
THE SUPPORTING INFORMATION NECESSARY TO BE 
CONSIDERED AT A GIVEN NODE 


" SIGNIFIGANCE 


e EACH BRANCH SHOULD REPRESENT AN IDENTIFIABLE 
AND MEASURABLE ELEMENT OF CONTRIBUTING 
INFORMATION 


FAMILIARITY 


e EACH BRANCH SHOULD BE UNDERSTANDABLE TO THE 
EXTENT IT IS POSSIBLE TO ESTABLISH A WEIGHTING 
RELATIVE TO THE OTHER BRANCHES AT THE SAME 
NODE 

INDEPENDENCE 

@ TWO BRANCHES FROM THE SAME NODE SHOULD NOT 

— REPRESENT THE IDENTICALLY SAME INFORMATION 
380476 24 (627.83) REQUIREMENT 

— BE DEPENDENT ON EACH OTHER 


In developing an information value tree, there are some simple 
analytical guidelines for delineating the set of branches at a particular 
node. Since each node represents the tieing together of all its relevant 
data, the set of branches must be “complete”, that is there must be a branch 
for each contributing piece of information. For example, “immediate 
affordability” is completely defined by what the down payment and the closing 
costs are. By the same reasoning, a branch should only be constructed if it 
represents a “significant™ element of information, that is, it has some 
relevance to the information requirement represented by the node. The fact 
that the house has a tile roof is not relevant to whether it is big enough. 
Next, each branch should be “familiar”, that is, it should represent a piece 
of information that both the analyst and reviewer can understand and 
evaluate. Last, all the branches should be “independent”; no two branches 
should depend directly on one another or represent the identically same piece 
of information. 
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DELINEATION OF INFORMATION TREE WEIGHTS 


DELINEATION OF WEIGHTS 


JOHN HAS NEVER BOUGHT A HOUSE BEFORE, SO 
HE IS NOT SURE WHAT THE RELATIVE 
IMPORTANCE OF THE VARIOUS ISSUES ARE 


TO GAIN INSIGHT, HE CONSULTS SOME OF HIS FRIENDS 


(WHO HE BELIEVES TO BE KNOWLEDGEABLE IN THE AREA -- 
LOCAL EXPERTS) 


MARY: WHO IS IN THE PROCESS OF BUYING HER THIRD 
HOME 


SUE: WHO HAS EXTENSIVE REAL ESTATE 
INVESTMENTS 


GARY: WHO HAS NEVER BOUGHT A HOUSE, BUT WHO 


JUST FINISHED A SEMINAR FOR FIRST-TIME 
HOME BUYERS 


26047525 (627 83; 


Once the structure of the information value tree has been established 
(the nodes and the branches have been defined), the next step is to delineate 
the relative value weightings. This can be done by the decision maker, in 
this case, John, or it can be delegated by the decision maker to a group of 


experts, 


Since John has virtually no experience in the real estate market, he 


decided he needed help in assigning the weights to the branches of his tree, 
So he selected three experts to advise hin. 


Mary is in the process of buying her third home so she's had a fair 
amount of experience not only in actually buying a house, but also 
in maintaining one. Of all his “experts”, she probably represents 
the one whose experience John feels is the most consistent with his 
perspective. 

Sue is very much involved in the real estate market and has many 
investments. She is predominately interested in investment 
potential, 

Gary has never bought a house but he just finished attending a 
seminar designed for first-time home buyers so he has information 
about what the “experts” say should be considered when buying a 
house, 
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The weight assigned to each branch reflects its value or contribution to 
the information requirement relative to the other branches emanating from the 
same node. This weight may range in value between 0 and 1 where 


0 = - the piece of information has no impact on the information 
requirement at all, and 

1 = this piece of information is the only contributing information 
required. 


The weights assigned to all the branches at a particular node must sum to l, 
reflecting the premise that together they represent the complete set of 
significant information needed to define the information requirement at that 
node. : 

The actual calculation of the branch weights is a two step process. 
First, the relative value of all the branches at a particular node is 
established. For example, piece B of information may be twice as important as 
piece A and piece C of information may be three times as important as piece 
A. This can be analytically described by: 

3*B = 6%A = 2*C 
The next step is to normalize by forcing the weights to sum to l which gives: 

At+B+tC =A t 2%A + 3*A = 1 


or 
= 1/6 : 
1/3 
172 

By requiring the branch weights to sum to 1 at every node, a consistent 
value structure is forced on the entire tree, enabling the analyst to readily 
calculate the relative value of any piece of information or data item within 
the tree. Ultimately, this leads to a preference ranking over the set of 
collectables (data items) which can then be used to evaluate the set of 
potential collection concepts. 


QP 
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ASSIGNMENT OF RELATIVE VALUES 


PROCESS WHEREBY THE RELATIVE VALUE OF THE 
VARIOUS PIECES OF INFORMATION (BRANCHES) 
REQUIRED TO SUPPORT A PARTICULAR (HIGHER) 
LEVEL OF INFORMATION (NODE) ARE ESTABLISHED 


‘VALUES (OR WEIGHTS) MAY RANGE BETWEEN 0 AND 1 
WHERE 


0 — HAS NO IMPACT AT ALL 
1 — IS THE ONLY CONTRIBUTING FACTOR 


AT A PARTICULAR NODE (QUANTUM OF INFORMATION), 
THE WEIGHTS ASSIGNED TO ALL THE CONNECTING 
BRANCHES (SUPPORTING INFORMATION) MUST SUM 
IDENTICALLY TO 1 TO ALLOW COMPARISON OF 


INFORMATION REQUIREMENTS AT NODES OF THE SAME 
AND HIGHER LEVELS 


200479 26 1627-83) 


CALCULATION OF BRANCH WEIGHTS 


GOALS: 


e THE WEIGHTS REFLECT THE RELATIVE VALUE OF THE SUPPORTING 
INFORMATION TO DEFINING THIS PIECE OF INFORMATION 


e ALL THE WEIGHTS CORRESPONDING TO THE SUPPORTING 
INFORMATION MUST SUM IDENTICALLY TO 1 


PROCESS: 


e ESTABLISH THE RELATIVE VALUE OF THE VARIOUS INFORMATION 
REQUIREMENTS, EG 
VALUE (B) = 2 VALUE (A) 
VALUE (C) = 3eVALUE (A) 
OR 
3B = 6A =2C 


e NORMALIZE TO FORCE WEIGHT TO SUM TO 1, EG (AS, ABOVE) 
A+B+C=1 
A+2A+3A=1 
6A =1 
A=1/6,B=1/3,C=1/4 
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SAMPLE OF EXPERT OPINIONS 


BIG ENOUGH _ BIG ENOUGH 


LOCATION LOCATION 
0.70 0.50 
MAINTAIN MAINTAIN a MAINTAIN 


0.20 i 0.30 - : 0.50 
OURABILITY OURABILITY OuRABILITY 


0.46 = 0.20 0.40 
IMMEDIATE IMMEDIATE imMEDIATE 


0.45 0.20 0.40 
INVESTMENT] CONTINUIN INVESTMENT) CONTINUING INVESTMENT[ CONTINUING 


26047926 (6 27.83) 


Three very different sets of weights were provided by John's experts as 
shown in the trees above. Mary and Sue, for example, have divergent 
priorities for the “investment potential" in general and the “future worth" in 
particular, Mary is much more concerned about the supportability of the house 
and the functionality it provides; Sue, with her background in real estate 
investments, concentrates on the investment section of the tree. Gary has yet 
another set of weights although his are closer to Sue's. 

The next issue for John is how to aggregate these three different ~ 
perspectives into a single consensue tree. , 
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The derivation of consensus weights for the decision maker's tree 

requires some procedure for aggregating the values for the weights provided by 
experts into a single set of estimates. Just summing the values up and taking 
the average, although a simple approach, does not take into account any 
information the decision maker may have about the differing levels of 
expertise the experts may have or. any knowledge of their individual biases. 
Additionally, such a procedure provides no way to adjust the impact extreme 
opinions may have on the aggregation (either to emphasize or to mask them). 

The question of how best to create a consensus position from a pool of 

expert opinions has received much attention from decision analysis experts. 
Many procedures have been proposed varying from the simplistic, such as the 
averaging approach suggested above, to the theoretically sophisticated; as 
yet, however, there is no general agreement about which approach yields the 
most valid results. However, to be useful for the collection systems problen, 
an aggregation procedure should meet four requirements: 

(1) There should be a way to incorporate any preferences the decision 
maker may have between the experts so differing levels of emphasis 
can be assigned in proportion to the perception of their expertise. 

(2) In order to preserve the relative value structure of the entire 
tree, the aggregation procedure must produce consensus weights such 
that the sum of the values assigned to the set of branches at any 
node is still exactly l. 

(3) The weight structure for any particular expert must be incorporated 
in such a way that his or her relative rankings are preserved. 

(4) To enable the procedure to be as usable as possible, the 
information required to implement it should be readily available. 

An example of one of the more analytic approaches that has been 

developed is a procedure proposed by Morris. It is a very rigorous Bayesian 
approach that uses each expert's past performance to condition, that is 
weight, his or her current prediction. Initially, the excellent theoretical 
foundation for this procedure, as well as others of a similar nature, seemed 
to strongly point to the implementation of such an approach within this 
methodology. Upon further investigation, however, the very strength of such 
procedures proved to also be their weakness, at least relative to the type of 
decision support methodology that is needed to handle the collection system 
problem. The application of approaches such as Morris's requires access to a 
reasonably large sized data base describing the performance of the expert with 
similar problems. Typically, such a data base is not available. As a result, 
the approach selected for implementation with this methodology is less 
sophisticated but in return the decision maker will have more control of how 
the consensus is to be achieved. If over time the data base necessary to 
support a method such as Morris" can be accumulated, a more sophisticated 
approach can be added to the procedure suggested in this report. 
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AGGREGATION OF EXPERT OPINION 


COMBINING THE OPINIONS OF SEVERAL EXPERTS IN A 
VALID MANNER ONE NEEDS TO CONSIDER 


@ DIFFERING LEVELS OF EXPERTISE BETWEEN 
EXPERTS 


@ INDIVIDUAL BIASES 
_@ EFFECTS OF EXTREME OPINIONS 


METHODOLOGY CHOSEN MUST ALLOW FOR THE 
ADJUSTMENTS NECESSARY AND ALSO BE 
SATISFACTORY IN THE 


@ TEMPORAL 


@ REASONABLE IN THE AMOUNT OF DATA 
REQUIRED 


@ EASILY TRANSFERABLE AND AT THE SAME TIME 
MAINTAIN AS MUCH DATA AS POSSIBLE 


MORRIS APPROACH 


APPROACH TO EXPERT RESOLUTION OR 
CONSENSUS DISCUSSED IN THE PROPOSAL 


RIGOROUS BAYESIAN APPROACH 


BASED ON PRIOR PERFOMANCE OF EXPERTS 


DRAWBACK 


— DATABASE ON PRIOR PERFORMANCE NOT 
TYPICALLY AVAILABLE 


APPROACH SELECTED 

— LESS TECHNICAL 

— GIVES THE DECISION MAKER MORE DIRECT 
CONTROL OF THE CONSENSUS PROCESS 
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For the collection systems problem, the approach selected to derive the 
consensus estimate from a set of expert opinions is based on a weighted 
average of the experts’ opinions. Given the estimates of N experts for a 
value to be assigned to a particular weight, 

V(1), V(2), «2+, VON), 
as well as a set of preferences (or weights) from the decision maker 
associated with each of these experts, 

WC1), W(2), ~.-, WN), 
the consensus value (W*) is the normalized weighted average of the experts’ 
opinions, that is, the weighted average divided by the sum of the weights, 


at $ W*V]. 
glo ge 3 


This approach, besides being very straightforward to apply, is appealing in 
the way it allows the decision maker to emphasize the estimates of the various 
experts in a way that is consistent with his or her evaluation of each of the 
experts. Another benefit is that it does not require the extensive historical 
database needed to implement an approach like Morris's. In fact, if the 
decision maker has no preference between the experts, the weights, Wj, can 

be set identically to l. , 

John has decided that of his three experts, Mary's perspective is the 
one that most closely matches his position. As such, he feels that her 
estimates should be weighted about twice as much as either Sue's or Gary's, 
whose opinions he feels should be weighted about equally. The resulting 
preferences or weights are 


Mary: 2 
Sue: 1 
Gary: 1 : 


Using these values, John calculates the consensus weights for his information 
value tree. As an example, the expert opinions associated with the branch for 
“functional needs” are 

Mary: 0.35 

Sue: 0.15 

Gary: 0.25 
giving a consensus weight of 

We = (1/4)*((2)(0.35) + (1)(0.15) + (1)(0.25)] 

= 0.275 

The complete set of consensus estimates for the weights are shown in the 
adjoining table. 
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CONSENSUS METHODOLOGY 


GIVEN 


e A SET OF WEIGHTS OR RELATIVE VALUES PROVIDED 
BY N EXPERTS, 
Vi... VN 


e A SET OF DECISION MAKER PREFERENCES OR 
WEIGHTS ASSOCIATED WITH THOSE N EXPERTS, 


Wy. erry Wy 
THEN 
e THE CONSENSUS WEIGH W*, 1S DEFINED BY 


EXAMPLE . 
e IF V4 =0.35, Vo = 0.15, V3 =0.25 
Wy = 2, W2=1, W3=1 
e THEN W*=1 (2) (0.35) + (1)(0.15) + (1) (0.25) = 0.275 
4 


380678 29 (6-27.63; 
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FUNCTIONAL NEEDS 
BIG ENOUGH 
BEDROOMS 
SQ FOOTAGE 
LOCATION 
SCHOOLS 


ee eee | CALCULATION 
SDeGRT REGHIAED oe learlea cs OF 
Morag por bet ella CONSENSUS 


CONDITION OF SIMILAR : WE | G HTS 


DISASTER VULNERABILITY 


INVESTMENT POTENTIAL 
IMMEDIATE AFFORDABILITY 
, OOWN PAYMENT 
CLOSING COSTS 
CONTINUING AFFORDABILITY 
BASIC MONTHLY 
INTEREST 
PROJ. ASSESSMENTS 
FUTURE WORTH 
EXP. APPRECIATION 
PLANNED DEVELOPMENT 
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The consensus tree developed by John is shown on the adjoining page at 
the left in the top figure. As required, the weights associated with the 
branches at each node sum to 1. John has reviewed this tree and is fairly 
comfortable with both the structure and the weights. 

On the right of the figure is another version of the same consensus tree 
which shows the relative value of each piece of information to all other items 
at the same level instead of just those items at the same node. The 
“maintainability” issue, for example, is evaluated to be three times as 
important as the question of the house's “future worth”. For this version of 
the tree, the values are calculated by multiplying the series of branch 
weights from the original tree along the tree from the decision maker question 
to that specific branch. In the case of the “maintainability” branch, the 
weight is calculated by multiplying the weight for that branch times the 
weight for the “support” branch, that is, 

W = (0.700)*(0,376) 
= 0.2622 

This approach to structuring the information value tree provides a 
ranking at each level of all the corresponding information requirements. This 
is particularly useful at the data item level as it produces a prioritization 
of the collection requirements. In John's tree, the most important data item 
to be collected, the branch with the high relative value is the “age of the 
house” (0.164); at the other end of the spectrum, the least important data 
item is “possible assessments” (0.028). This modified form of the tree makes 
it readily apparent that the chief issue seems to be the support of the house 
and, more specifically, the maintenance of the house, whereas the least 
concern is given to its future worth. 

This completes the first step of the analysis, the development of an 
information value tree. The decision maker now has insight into the data 
items that would be desirable to collect as well as their relative 
importance. This ranking will be useful in the next step, the evaluation of 
the various candidate collection concepts. 
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JOHN’S CONSENSUS TREE 


BEDROOMS BEDROOMS 
SQ. FOOTAGE SIG ENOUGH eS 5a. FOOTAGE 


0.168 
FUNCTIONAL FUNCTIONAL 
Lal 
—e SCHOOLS ; ae SCHOOLS 
LOCATION WORK LOCATION WORK 


0.426 SHOPPING 0.117 SHOPPING 


8IG ENOUGH 
0.696 


a YARD YARD 
INA 
MAINTARNABILTY AGE.GE HOME MAINTAINABILITY 
BUY? |. SUPPORT : BUY? | SUPPORT ‘ 
1.000 0.376 CONDITION 7.000 0.376 CONDITION 
QURABILITY DURABILITY 


DISASTER tSASTER VUL. 
0.300 SASTER VUL 0113 OtSAS U 


AGE OF HOME 


DOWN DOWN 
IMM. AFFORD. : IMM. AFFORO 
0.376 CLOSING ; 0131 CLOSING 


SASIC MONTHLY BASIC MONTHLY 


INVESTMENT] CONT. AFFORO | INTEREST VAR. a INVESTMENT] CONT. AFFORD | INTEREST VAA. oa 
POSS. ASSESSMENT 0.360 0.431 POSS. ASSESSMENT 
0.21 ° 


0.360 0.375 


‘ EXP. APPRECIATION Gage EXP. APPRECIATION 7 
FUTURE WORTH < FUTURE WORTH . 
0 250 PLANNED O€EV. re o 0.088 PLANNED DEV. 


BRANCH VALUES BRANCH VALUES 
RELATIVE TO NODES RELATIVE TO EACH LEVEL 
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PRELIMINARY CONCLUSIONS 


@ CHIEF ISSUE 


~ MAINTAINABILITY OF THE HOUSE, SPECIFICALLY 
AS IT PERTAINS TO ITS AGE 


@ MOST IMPORTANT DATA ITEM 
— AGE OF HOUSE 


LEAST IMPORTANT DATA ITEMS 


POSSIBLE ASSESSMENTS 
LOCATION RELATIVE TO SHOPPING 
— LOCATION RELATIVE TO WORK 
EXPECTED APPRECIATION 
DISASTER VULNERABILITY 
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III. C. COLLECTION CONCEPT DEFINITION AND DATA DEVELOPMENT 


INFORMATION COLLECTION 
CONCEPTS 


‘NEXT STEP: ASSESS THE PERFORMANCE (1.E., THE VALUE 
OF THE INFORMATION COLLECTED) OF THE VARIOUS 
INFORMATION COLLECTION CONCEPTS TO BE STUDIED 


ACCOMPLISHED BY ASCERTAINING EACH CONCEPT’S 
ABILITY TO COLLECT EACH REQUIRED DATA ITEM IN 
TERMS OF 


e EXPECTED LIKELIHOOD OF COLLECTING THE 


REQUIRED DATA 
e THE MINIMUM LIKELIHOOD OF COLLECTION 
e THE MAXIMUM LIKELIHOOD OF COLLECTION 


WHERE THE LIKELIHOOD OF COLLECTION !S A FUNCTION 
OF 

e THE PROBABILITY OF COLLECTING THE DATA 
- @ THE QUALITY OF THE DATA COLLECTED, 1.E., THE 


ABILITY TO SUPPORT THE SUCCEEDING INFORMATION 
REQUIREMENTS 


36047932 (627-83) 


The next step of the methodology focuses on the evaluation of the 
ability of each candidate concept to meet each raw data collection — 
requirements. This, combined with the relative weight structure derived in 
the previous step, will provide a means for analyzing collection performance, 
For each concept, the collection ability relative to each data 7 
requirement will need to be specified not only in terms of its expected or 
likely performance, but in order to be able to do confidence assessments, 
minimum and maximum performance levels will also need to be delineated. All 
| performance likelihoods are a combination of both the probability of 
collection and the quality of what is collected relative to the data 

requirement. For example, relative to information about “planned 
developments” in the local area, if one of John's candidate concepts on the 

| average will have an 80% chance of collecting some data, and if collected, the 
data will probably cover 95% of the developments, then a likely performance 
level, P, would be 

| P= .80 * ,95 

or approximately 75%. Quality thresholds, i.e., stipulation that if the data 

is not at least a certain quality it is not any good at all, are handled by 
| assigning a 0 to the quality factor if it falls below the threshold. 
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John has identified four candidate approaches to collect the data he 

needs: 
. (1) Drop by the house in question and talk with the owner. 

(2) Call up the city for information. 

(3) Arrange with a friend (who is a real estate agent) to get the 

necessary data. 

(4) Check the newspaper to see if the owner is advertising the house. 

There may be other approaches that he might use, but these four are the ones 

under consideration. The question then is which one is the most effective 

collection concept, which one will collect the information with the highest 

overall potential value. 

The first step in evaluating any of these collection concepts is to 
define the concept's ability to collect each data item in terms of its best or 
maximum performance, its worst or minimum performance, and its likely or 
expected performance. Data on the performance of the concepts can be provided 
by the decision maker or by an expert or experts designated by the decision 
maker. In the case of more than one expert, the same consensus techniques 
used to delineate the branch weights can be applied. 

The table on the adjoining page summarizes the performance data for 
John's problem for each of his collection concepts, For example, in the area 
of “disaster vulnerability”, John's estimate is that the owner's likelihood of 
supplying him that information is somewhere between 40% and 60%, the city's 
likelihood is between 60% and 80%, and his friend's likelihood is ‘between 50% 
and 70%. He does not expect to find this particular data item in a newspaper 
advertisement. 
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POTENTIAL SOURCES FOR 
JOHN’‘S DATA 


JOHN HAS IDENTIFIED 4 POTENTIAL WAYS TO COLLECT 
THE DATA HE NEEDS: 


(1) DROP BY THE HOUSE IN QUESTION AND TALK WITH 
THE OWNER 


(2) CALL UP THE CITY FOR INFORMATION 
(3) ARRANGE WITH A FRIEND WHO IS A REAL ESTATE 
' AGENT TO GET THE NECESSARY DATA 


(4) CHECK THE NEWSPAPER TO SEE IF THE OWNER IS 
ADVERTISING THIS HOUSE 


HE NOW NEEDS TO DEFINE THE PERFORMANCE OF EACH 
OF THESE RELATIVE TO EACH DATA REQUIREMENT IN 
TERMS OF MINIMUM, EXPECTED, AND MAXIMUM 
PERFORMANCE ‘ 


e CAN BE DONE BY THE DECISION MAKER OR BY USING 
EXPERT CONSULTATIONS AND CONSENSUS 
TECHNIQUES AS WAS DONE WITH THE WEIGHTS 


360478-23 (6-27.83) 


INFORMATION PERFORMANCE [bs (uucuee’ 
FACTORS  — fbr 


TALK WITH 
FRIENO 


NUMBER OF BEDROOMS 
SQUARE FOOTAGE 
SCHOOLS (LOCATION) 
WORK (LOCATION) 
SHOPPING (LOCATION) 


AMOUNT OF YARD WORK 
AGE OF HOME 

CONDITION OF SIMILAR HOMES 
DISASTER VULNERABILITY 


DOWN PAYMENT 
CLOSING COSTS 
BASIC MONTHLY PAYMENT 
PROJECTED INTEREST VARIANCE 
POSSIBLE ASSESSMENTS 
EXPECTED APPRECIATION 
PLANNED DEVELOPMENTS 
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Once the information has been gathered on the performance range of each 
candidate collection concept, the next step is to evaluate their expected 
performance. For each information requirement, this is expressed in terms of 
an expected information value which is determined by multiplying the expected 
performance level by the corresponding relative weight from the modified 
information value tree. In the case of the third collection concept (talking 
with his friend) and the first collection requirement (the number of bedrooms), 

relative value of information = 0.075 

expected performance = 0.9 

expected information value = (0.9)*(0.075) 
= 0.0676 
The expected information values are in turn summed over the entire tree to 
provide the overall expected information value for the complete tree for each 
concept. These can be compared to determine the collection concept with the 
highest expected performance. 

The adjoining table contains the expected performance figures for each 
of the collection concepts in John's problem. The first and the third 
concepts (talking with the owner and talking with his friend) have markedly 
higher expected performance than the remaining candidates. The margin between 
the top two, however, is close enough that it is difficult at this point to 
differentiate between them; not enough is known, only the expected performance 
has been compared, not the complete performance range. To do this, John will 
need to move into the third phase of the evaluation provess, the sensitivity 
analysis. 
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PERFORMANCE ANALYSIS 


THE INITIAL CONCEPT ANALYSIS IS BASED ON THE EXPECTED 
PERFORMANCE OF EACH CANDIDATE COLLECTION CONCEPT 
AND USES 


— EXPECTED PERFORMANCE LIKELIHOODS 
"_ CONSENSUS WEIGHTS 


BASICALLY INVOLVES WEIGHTING THE EXPECTED 
PERFORMANCE LIKELIHOODS WITH THE WEIGHTS (RELATIVE 
TO THE DATA ITEM LEVEL) ASSOCIATED WITH THE 
INFORMATION REQUIREMENTS 


‘weeee.2 (7-21-43) 


EXPECTED PERFORMANCE HUGHES | 


Re weeee eee) 


= 
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SENSITIVITY ANALYSIS 


SENSITIVITY ANALYSIS 


@ SENSITIVITY ANALYSIS !S THE CRUCIAL STEP OF THE 


METHODOLOGY 


@ PROVIDES DECISION MAKER WITH THE 
QUANTITATIVE LIMITS OVER WHICH HIS/HER 
QUALITATIVE DECISION WILL BE VALID 


. ANALYSIS CAN BE APPLIED SEVERAL PLACES 


@ OVERALL TREE 

@ INTRALEVEL ON THE TREE 

@ BETWEEN CONCEPTS 

@ INTERNAL TO CONCEPT ANALYSIS 


METHODS 


_@ WORST CASE/BEST CASE 


@ MONTE CARLO 


The sensitivity analysis is perhaps the most critical phase in the 
methodology because it provides the decision maker insight into the limits 


over which the collection concept selected is the most effective. Finding the 
concept with the highest expected value for the information collected is only 
the first part of the solution; it is equally important to understand the 
range over which it is the most effective solution and how sensitive the value 
of the information collected is to variances in each concept's performance 
relative to the various information requirements. 

Classically, sensitivity analysis refers to procedures in which 
parameters are varied over their possible values in order to determine the 
degree of variation in the resulting solution. Here sensitivity analysis will 
be used in two ways 
to evaluate the impact on the overall information value over the 
entire range (not, just the expected value) of each collection 
concept's performance relative to the various information 
requirements, and 
to incorporate factors in addition to the value of information that 
differentiate the performance of various collection concepts. 


Note: The reader is once again warned that the example illustrated here has 
been specifically designed to illustrate all aspects of the developed 
methodology. In actual practice, not all the steps shown will be 


necessary as each problem will possess possibilities for simplification. 
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JOHN’S PERFORMANCE 
ANALYSIS 


TALK WITH | TALK WITH FRIEND CHECK 
OWNER CITY (AGENT) NEWSPAPER 
Lmin fax | Min {wax | min {wax 1 m0 | MAX | 
NUMBER OF BEDROOMS . 
SQUARE FOOTAGE 
" SCHOOLS (LOCATION) 
WORK (LOCATION) 


SHOPPING (LOCATION) 


A a a Pans [ors | 
as? [oss | 020] 026 | os7[osa | oa [are | |] 
CONDITION OF SIMILAR Aseofrowe — 56 | 0.59 0.59 | 0.62 
oa i ee 
DOWN PAYMENT 
CLOSING COSTS 
BASIC MONTHLY PAYMENT 
PROJECTED INTEREST VARIANCE 
POSSIBLE ASSESSMENTS 


EXPECTED APPRECIATION 
PLANNED DEVELOPMENTS 


260476 36 (6-27-63) 


The first step of a more in-depth analysis of the performance of the 
various candidate collection concepts involves investigating the changes in 
the overall value of the information collected at the limits of the 
performance ranges associated with the various information requirements. The 
purpose of such an analysis is to determine 

= the impact each collection requirement has over its performance 

range on the value of information collected, and 

- the degree to which the various concepts overlap, that is, are- 

indistinguishable. 
The effect of the performance of each concept relative to each information 
requirement is studied by re-calculating the overall value of the information 


collected by 
= fixing the performance of the requirement under consideration first 


at its minimum and then its maximum performance level, and 
- using the expected value performance estimates for all other 
information requirements. 
This provides the analyst insight into the factors that determine which 
concepts are the most effective. 


a | 
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The results of such a performance analysis are summarized in the 
adjoining table. Each entry represents the impact relative to a particular — 
collection concept of evaluating a specific information requirement at either 
the minimum or maximum (versus expected) level of concept performance. There 
are a couple of observations of note. 

- The impact of some of the information requirements seems to be 
independent of whether collection concept performance is fixed at 
its minimum, expected, or maximum level of performance. Examples 
are: 

- number of bedrooms 

- location relative to work 

a disaster vulnerability 

- down payment 

= basic monthly payment 

- projected interest variance 

- possible assessments 

= expected appreciation 
When this occurs, a more detailed analysis to determine the exact 
way in which collection concept performance varies between its 
minimum and its maximum levels is unnecessary; it's adequate to fix 
its performance at the expected level. 
The four candidate concepts have different ranges of information 


value: . 
(1) Talking with owner 56-59% 
(2) Talking with the city 20-26% 
(3) Talking with friend 57-63% 
(4) Checking newspaper 12-18% 


The range of values for concept #2 and for concept #4 in no way 
overlaps that of any of the other concepts. However, concept #1 and 
#3 overlap considerably. Usually talking with the friend seems to 
have a higher return than talking with the owner, but not always 
(see “age of house” for example). 
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INCORPORATION OF OTHER FACTORS 


THE EVALUATION OF COLLECTION CONCEPT PERFORMANCE 
MUST INCLUDE TWO OTHER FACTORS 

e TIMELINESS 

e SURVIVABILITY 


‘ TIMELINESS 


@ THE IMPACT ON THE VALUE OF THE INFORMATION 
COLLECTED DUE TO THE DELAYS IN IMPLEMENTING 


THE CONCEPT OR DELAYS IN THE CONCEPT OF 
PRODUCING THE DATA 


SURVIVABILITY 


e THE IMPACT ON THE VALUE OF THE INFORMATION 
COLLECTED DUE TO THE LIKELIHOOD OF THE 
COLLECTION CONCEPT ENDURING OR SURVIVING 


BOTH ARE FUNCTIONS OF EITHER 


e THE DECISION MAKER’S PERCEPTIONS, OR 
e THE CONSENSUS RESULT OF GATHERING THE 
OPINIONS OF EXPERTS SELECTED BY THE 
App ie DECISION MAKER 


The evaluation of collection concept performance must include two other 
factors: timeliness and survivability. While the overall value of the 
information collected is the key discriminant between concepts, that value can 
be impacted by delays in implementing it or by the length of time over which 
it continues to provide information. The extent of that value impact is a 
function of the performance of the concept as well as the decision maker's 
evaluation of its importance. 

A key concern in collection system analyses is the incorporation of. 
revisit times or capabilities into the analyses. Revisit capabilities are 
accounted for in one of two ways in the methodology. The first approach, 
which is the most straightforward and easily applied, involves incorporating 
revisits into the collection concept performance assessment. With this 
approach the concept is viewed over time with the assessment of capability 
including several collection time intervals or opportunities. In simple 
terms, the question to be answered would be: Given the concept will visit 
this often and have these attempts at collection, what is the probability you 
will satisfy this particular information need? In this sense the concept 
capability includes performance over time and the details of individual visits 
or opportunities are suppressed in favor of an overall performance assessment. 
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The second alternative which can be applied to revisit questions 
involves using discounting methods similar to those discussed under concept 
survivability. In this approach each opportunity is analyzed individually and 
multiple opportunities over time are combined using the discounting methods 
discussed later in this section (see page 46). The application of this method 
involves selecting an appropriate discount factor. 

The selection of either the overall performance assessment or the 
discounted assessment will depend upon analyst confidence and knowledge. The 
ultimate goal, however, is an assessment of overall performance which the 
analyst feels is believable and accurately reflects systems capabilities. 
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TIMELINESS 


e GIVEN 
V=VALUE OF INFORMATION NOW 
N=AMOUNT OF TIME BEFORE IT CAN BE COLLECTED 
Ft(N)=DISCOUNT FACTOR TO BE APPLIED TO 
INFORMATION VALUE GIVEN A DELAY OF N 
Pt(N)=PROBABILITY OF A DELAY OF N 


e THEN 
Et (V)=EXPECTED VALUE OF INFORMATION GIVEN 


TIMELINESS CONSIDERATIONS 


=V>Ft (j) * Pt (i) 
j 


380479 6916 27 63: 


When there is a gap between the point in time the need for some 
information is assessed and when it is actually received, the value of what is 
collected may be reduced. To evaluate collection concepts with different 
delays, the decision maker must specify to what extent delays of various 
durations impact the overall value of the information collected. This impact 
is expressed as a discounting factor on the overall value of information. 


Given 
V = overall value of information relative to some decision 
maker question 
N = number of intervals that the collection of that 


information is delayed 


FeCN) = timeliness discount factor associated with a delay of N 
intervals 
then 
V_ CN) = discounted value of information given a delay of N 
. intervals 
= VeFe(N) 


Typically, the discount factors, F,_(N), are expressed as elements ina 
discount stream, i.e., if each period of delay corresponds to “p” percent 


loss, then 
1 N 
F.C mu) Fea 
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For example, a 5% discount factor for 4 periods would be 


1 4 
oa as [as 
= 0.8227 
Since the decision maker may not be able to predict a priori the exact 
number of intervals that the receipt of information may delayed, there is 
usually a need to specify the probability of delays of various duration and to 
evaluate the expected discounted value of information. 


Given. 
Pe(N) = probability of a delay of N intervals 
then 
Er (CN) = expected value of information discounted for timeliness 
M 
= E [VeC5) * Pe(5)] | | 
j=l 
= ve Z [Fe(j) * Pe(5)] 
j=l 


Note that the probability distribution does not need to be specified in a 
detailed way. It is sufficient to grossly estimate it over a handfull of 


points; for example, 


P+(0) = 0.2 
Pr(1) = 0.4 
P+(2) = 0.3 
P-(3) = 0.1 
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SURVIVABILITY 


e GIVEN 
V=VALUE OF INFORMATION NOW 


_N=AMOUNT OF TIME OVER WHICH IT WILL BE 
COLLECTED 
Fs(N)=ENHANCEMENT FACTOR TO BE APPLIED 


FOR INFORMATION IN THE Nth PERIOD 
Ps(N)=PROBABILITY OF AN ADDITIONAL N PERIODS 


e THEN 
Es(V)=EXPECTED VALUE OF INFORMATION GIVEN 
SURVIVABILITY CONSIDERATIONS 


k 
=v* EY P (k)* [= F_(i)] 
k s j=0 S 


When information is collected over an extended period of time, the value 
of what is collected may be enhanced. To evaluate collection concepts with 
different survivabilities, the decision maker must specify to what degree 
extended collection enhances the overall value of the information collected. 
This impact is expressed as a compounding of value over the number of 


intervals. 
Given . 
Vv = overall value of information relative to some decision 
maker question 
N = number of intervals over which the collection of that 
information continues 
Fs(N) = relative value of some information in the Nth interval 
then 
VsCN) = enhance value of information over N intervals 
N 
= 2 valk 5( 4) 
j=0 


In other words, the decision maker must specify the incremental benefit of 
each additional interval of collection. If there is none, then F,(j) is set 
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to zero. As with timeliness, however, these compounding factors are typically 
expressed as elements in a discount stream, i.e., if each interval of extended 
collection corresponds to a “p” percent discount factor, then 


1,N 
F.( N) = ra 
Since the decision maker may not be able to predict a priori the exact 
number of intervals over which the receipt of information may continue, the 


probability of survival over various durations usually must be specified and 
the expected enhanced value of information evaluated. 


Given 
Ps(N) = probability of surviving N intervals 
then 
E,(N) = expected value of information enhanced for survivability 


5 {¥scx) * Ps) | 


il 


N k 
vez fr * © tFa1} 
: k=0 j=0 - 
Again, the probability distribution does not need to be specified ina 
detailed way. It is sufficient to grossly estimate it over a handful of 


points. For example, 
if additional information is discounted at a rate of 102, and 


P5(0) = 0.4 
Paty =. 003 
P5(2) = 0.2 
P5(3) = 0.1 
then over 3 years 
2 eee eee 
B@)- ve E {rc * 5 (Eq I] 
k=0 j=0 ‘ 
1 1 1 
=V* (4x1) + 3 (14-7) + .2* (try t —o » 
€le2) 
+.1* 1+ +. + 5+ ule 3| 
° (1.1) lel - 
= 1.8685 * V ( ) 
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IN ORDER TO GET THE INFORMATION FROM HIS REAL 
ESTATE FRIEND, JOHN WILL HAVE TO WAIT AT LEAST 
TWO WEEKS UNTIL HE COMES BACK FROM VACATION 


JOHN FEELS THAT THIS INEFFICIENT USE OF TIME WILL 
REDUCE THE VALUE OF THE INFORMATION HE COLLECTED 
BY THE FACTORS 


1 
7.050 ~ -9 


ALL OTHER CONCEPTS HAVE COMPARABLE 
TIMELINESS 
SURVIVABILITY 


THEREFORE THE PERFORMANCE ANALYSIS NEEDS TO 
BE RECONSIDERED 


3806 79-40 (6-27-03) 


In order to get the information from his friend in real estate, John 
will have to wait at least two weeks until he comes back from vacation. Over 
this period of time, John feels that the value of the information that will be 
collected will degrade somewhat due to the fact that the house may go off the 
market. Therefore he estimates that a delay of two weeks will discount the 
value of what is collected by about 5%, for a discount factor, F, of 

F = 1/1,05 = 0.95 
This third collection concept is the only one expected to have such a delay; 
John expects to be able to implement any of the others immediately. 

To adjust John's analysis to reflect this delay in timeliness for the 
third candidate collection concept, what is required is the reduction of all 
the relative values associated with the information requirements by 0.95 since 
by the time the information is collected it will have been discounted in value 
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by about 5%. For example, the expected information value the third collection 
concept is expected to collect for the first collection requirement, the 
number of bedrooms, is recalculated by 

relative value of information = (1/1.05) * (0.075) = 0.072 

expected performance = 0.9 

expected information value = (0.9) * (0.072) = 0.0648 
When all of the expected information values have been recalculated, they are 
again summed to determine the discounted expected value for the information 


collected. 
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The adjoining tables show the effect that the discounting had on both 
the performance factors associated with the third concept and on the 
performance analysis as a whole. The expected performance of the third 
concept, talking with the friend who is a real estate agent, typically is 
reduced between 3 and 4 percent across all of the information requirements. 
As for the comparison of the complete set of collection concepts, a check of 
the performance ranges shows that while concepts 2 and 4 are still much lower 
than 1 and 3, the choice between 1 and 3 is even less clear than before. 
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NUMBER OF BEDROOMS 
SQUARE FOOTAGE 

SCHOOLS (LOCATION) 
_WORK (LOCATION) 

SHOPPING (LOCATION) 
AMOUNT OF YARD WORK 

AGE OF HOME 

CONDITION OF SIMILAR HOMES 
DISASTER VULNERABILITY 


DOWN PAYMENT 

CLOSING COSTS 

BASIC MONTHLY PAYMENT 
PROJECTED INTEREST VARIANCE 
POSSIBLE ASSESSMENTS 
EXPECTED APPRECIATION 
PLANNED DEVELOPMENTS 


VALUE | VALUE 
| se | so | 
[20 | 6 
| sa | 
a 
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STOCHASTIC DOMINANCE 


Reece. oe aaaecieeca 


¢ A CONCEPT IS COMPLETELY DOMINATED IF THERE 
iS AT LEAST ONE OTHER CONCEPT HAVING ALL ITS 
MINIMUM INFORMATION VALUES GREATER THAN ANY 
MAXIMUM INFORMATION VALUE FOR THE 
DOMINATED CONCEPT 


e SUCH CONCEPTS NEED NO FURTHER 
CONSIDERATION 


3904 79-67 6-27 83) 


In analyzing the competing collection concepts, there occasionally will 
be a case where the performance of one of the candidate concepts is so low 
relative to other concepts that it no longer needs to be considered. A 
concept, A, is said to be stochastically or probabilistically dominated if no 
matter what values the collection performance variables take, there is at 
least one other concept, B, such that for every information requirement, Rj, 
the minimum performance range associated with B exceeds the maximum 
performance associated with A. Since the goal is to find the single most 
effective concept, no further consideration needs to be given to a dominated 
concept.* ¢ 

As shown in the preceeding table on performance ranges, concepts 2 and 4 
are completely dominated by concepts 1 and 3. Therefore John will no longer 
consider concepts 2 and 4 in his analysis. 


*Except if the ultimate use is in cost-effectiveness trade off. However, this 
is not the problem presently being considered. 
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The ultimate goal of this methodology should not be the achievement of a 
stochastically dominant solution. While this is desirable from the decision 
maker's viewpoint, his task is made easier, the non-linear aspects of the 
problem does not quarantee the existence of a dominant solution. Therefore, 
stochastic dominance is not used as a. goal for solution selection, but instead 
serves as a means of problem simplification by allowing for the identification 
and elimination of totally unacceptable solutions. If a single, dominant 
solution is not found, the decision maker must then view the decision in terms 
of required levels of performance, as well as risk aversion or acceptance 
preferences. Using these two measures, alternatives which do not dominate 
each other can be distinguished and a solution selected. 
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PROBABILISTIC ANALYSIS 


A MONTE CARLO APPOACH IS USED TO DETERMINE 
THE COMPLETE PERFORMANCE SPECTRUM FOR EACH 
‘CONCEPT 


PROCEDURE a 
— PERFORMANCE DISTRIBUTION RELATIVE TO EACH 


DATA REQUIREMENT ARE DEVELOPED FOR EACH 
NON-DOMINATED CONCEPT 

—~ GIVEN THE RELATIVE VALUE OF EACH DATA 
REQUIREMENT, THE PROBABILITY OF THE 
INFORMATION VALUE OBTAINED FOR EACH 
CONCEPT IS DETERMINED BY A MONTE CARLO 
ANALYSIS USING THOSE PERFORMANCE 
DISTRIBUTION 


— ALLOWS CONFIDENCE ASSESSMENT 


380479 63-627 62) 


To further compare the remaining concepts, it is necessary to get more 
insight into how the value of the information they collect varies over the 
performance ranges relative to each information collection requirement. This 
is done by developing distributions that describe the probability of achieving 
various values of information. These will allow the analyst to do confidence 
assessments, 

The first step in performing a probabilistic analysis is to roughly 
determine the performance distribution relative to each information 
requirement for each candidate collection concept still under consideration. 
The specification of these distributions can be a simple and as gross as 
merely specifying a handful of points (for example, the minimum performance 
level, the 10th percentile, 30th percentile, the 50th percentile (expected 
performance), 70th percentile, 90th percentile, and the maximum performance 
level). In John's example, all the performance distributions were assumed to 
be uniform, that is, given , 


a = minimum performance level 
b = maximum performance level 
- 54 - 
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then if P(v) is the probability of collecting information with at least a 
value of v, : 

P(v) = (v-a)/(b-a), as vsb 

P(v) = O, otherwise 
and if p(v) is the probability that the information collected had exactly a 
value of v, 

p(v) = I1/(b-a), asvesb 

p(v) = 0, otherwise 
Another distribution which also could have been selected is the normal 
distribution. The key, however, is that it is not necessary to specify a 
“classic” distribution if there is not one that represents the collection 
concept's performance relative to a particular information requirement; a 
simple estimate based on expert judgement is sufficient. 

It is important to note that while these performance distributions can 
be developed for every information requirement, those requirements which the 
performance analysis has shown to have virtually no impact on the overall 
value can simply be left at their expected value. This saves the analyst the 
extra work of defining distributions for those requirements. 

Once all the necessary performance distributions have been defined, they 
are used with Monte Carlo techniques to derive the probabilistic performance 
of the value of the information collected for each remaining candidate 
collection concept. The basic approach is to select samples from the 
performance distributions, weight them with the relative value of the 
associated information requirement, adjusted to reflect any timeliness or 
survivability considerations, and add them all together to get a sample of the 
distribution for the value of the information collected. More formally, 

given 

X5(1), the ith sample from the performance of jth information 

requirement, 

Va, the relative value of the jth information requirement, 

n, the number of information requirements 

Yy, the ith sample from the information value distribution 
then 


* X3(4) 


n 
v4 = > V3 
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The adjoining graphs illustrate the results of the probabilistic 
analysis for John's problem. The top figure is a graph of the probability of 
achieving a guaranteed value of information. Note that although the curves 
are close, there is a difference which influences which one should be 
selected. If John is risk-adverse, he should talk with the owner because 
there is less chance of getting the lowest possible values; if however John is 
risk-immune, he should instead talk with his friend because there is a chance 
he will collect a higher value. 
| The differences between the two concepts are even more clearly 
highlighted in the lower figure. Talking with the owner brings more 
'gonsistent results, there is less variance. Talking with a friend could just 
as easily give you a lower information value as a higher relative to that 

which can typically be achieved by the other concept. 


PRR EREREER SE E. 


Feel 


= 
_ 
ts 


SO 


Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1 


Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1 


PROBABILITY OF °° 
AT LEAST 
THIS VALUE 


0 Py Fs 
—— TALK WITH OWNER 0.50 0.55 ‘ 0.60 0.65 
— 7 TALK WITH FRIEND RELATIVE VALUE OF INFORMATION 


¥50479-5? (627-63) 


PERFORMANCE COMPARISON 


PROBABILITY 


TALK WITH OWNER oes 0.55 0.60 0.65 
TAROT RRIEND- ere RELATIVE VALUE OF INFORMATION 


0470-48 6-27-40) 
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DECISION MAKER REVIEW 


AT ANY POINT IN THE ANALYSIS, THE ENTIRE PROCESS 
ACCOMPLISHED TO THAT POINT IS COMPLETELY 
ACCESSIBLE TO 


@ REVIEW 
@ QUESTIONING 
@ RE-ASSESSMENT 


ALL ASSUMPTIONS ARE OUT IN THE OPEN AND THE 
ANALYSIS PATHS ARE READILY TRACEABLE 


380679-56 16-27-23) 


One of the strengths of the methodology is the generation of traceable 
paths through the analysis. Since every assumption is documented and the~ 
analytic process is straightforward and completely visible, the study is 
accessible at any point for review. This means the decision maker does not 
need to rely solely on an intuitive understanding of the problem which is 
especially important when the results are to be communicated and perhaps 
defended to others. The methodology itself provides a structure for 
discussing the analysis and the results and, if necessary, for defining at 
what point re-assessment may be necessary. Additionally, if re-evaluation is 
required, any new assumptions can be readily substituted for old with 
retention of much of the remaining structure, 
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To purchase a home, John is going to need financial assistance from his 
parents; so when he finished his collection concept analysis, 
his parents for review. Overall, they understood and agreed with what John 
had done. They did, however, feel that if this house was to be in a sense an 
investment for them as well, the weights on the information value tree should 


reflect more of an emphasis on the investment issues. 


he took it to 


Their recommendation 


was that the consensus tree should be recalculated, but this time with the 


following weights 
- 3 for Sue's values 


- 2 for Mary's values 
aie 1 for Gary's values 


The revised calculation of the consensus weights is shown in the 


table. The basic formula used was: 


W*(i) = ith consensus weight 


= (1/6) * [2 * (Mary's ith weight) + 
3 * (Sue's ith weight) + 
1 * (Gary's ith weight) ] 
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REVIEW OF JOHN’S ANALYSIS 


AFTER JOHN COMPLETED HIS ANALYSIS HE SHOWED IT 
TO HIS PARENTS SINCE HE NEEDS THEIR HELP TO 
FINANCE SUCH A PURCHASE 


IN GENERAL THEY AGREED WITH WHAT HE HAD DONE. 
-THEY FELT HOWEVER THAT THE WEIGHTS ON JOHN'S 
TREE SHOULD BE ADJUSTED TO REFLECT MORE OF THEIR 
EMPHASIS ON THE FUTURE WORTH OF THIS INVESTMENT 


THEIR SUGGESTION WAS THAT THEIR JOINT CONSENSUS 
TREE SHOULD BE CALCULATED WITH 


3 FOR SUE’S WEIGHTS 
2 FOR MARY’S WEIGHTS 
1 FOR GARY’S WEIGHT 


38047827 1627 63) 


RELATIVE 
GARY TO 
; TREE 
eavent raerenencesmecnts | 2 [s[1 | - | - 


FUNCTIONAL NEEDS 0.38 5 0.25 0.233 
BIG ENOUGH 
8EDROOMS 
SQ FOOTAGE 
LOCATION 
SCHOOLS 


WORK 40 |0.10/ 0. REVISED 
SHOPPING i .30| 0. j 
SUPPORT REQUIRED 50 [0.28] 0, CALCULATI ON 
MAINTAINABILITY ji ). .! . 
AMT OF YARD WORK ; : ; : ; O F 
AGE OF HOUSE , 70| 0. F 


ONDITIONIOE GHILAR ae lanl © CONSENSUS 


OISASTER VULNERABILITY 


INVESTMENT POTENTIAL Ww E I G HTS 


IMMEDIATE AFFORDABILITY 
OOWN PAYMENT 
CLOSING COSTS 
CONTINUING AFFORDABILITY 
BASIC MONTHLY 
INTEREST 
PROJ. ASSESSMENTS 
FUTURE WORTH 
EXP. APPRECIATION 
PLANNED DEVELOPMENT 
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COMPARISON OF DATA PRIORITIES 


AGE OF HOUSE 

AMOUNT OF YARD WORK 
SQUARE FOOTAGE 

CONDITION OF SIMILAR HOUSES 
NUMBER OF CEDROOMS 

DOWN PAYMENT 

CLOSING COSTS 

LOCATION RELATIVE TO SCHOOLS 
INTEREST VARIANCE 

BASIC MONTHLY PAYMENT 
PLANNED DEVELOPMENTS 
DISASTER VULNERABILITY 
EXPECTED APPRECIATION 
LOCATION RELATIVE TO WORK 
LOCATION RELATIVE TO SHOPPING 
POSSIBLE ASSESSMENTS 


AGE OF HOUSE 

PLANNED DEVELOPMENTS 
AMOUNT OF YARD WORK 
EXPECTED APPRECIATION 

DOWN PAYMENT 

CLOSING COSTS 

CONDITION OF SIMILAR HOUSES 
SQUARE FOOTAGE 

LOCATION RELATIVE TO SCHOOLS 
INTEREST VARIANCE 

BASIC MONTHLY PAYMENT 
NUMBER OF BEDROOMS 
LOCATION RELATIVE TO SHOPPING 
DISASTER VULNERABILITY 
POSSIBLE ASSESSMENTS 
LOCATION RELATIVE TO WORK 


390479-$4 (6-27-40) 


The revised consensus trees, both the basic (with branch weights 
relative to the preceeding node) and the modified (with branch weights 
relative to the level), are presented on the next page in comparison to the 
original consensus trees. Note how the emphasis has moved towards the 
investment issues in general and the future worth in particular. 

In the above table are ranked lists of collection requirements, one 
reflecting the original analysis, one reflecting this revised analysis. Some 
of the collection requirements have been significantly re-ordered, especially 

= square footage 

- number of bedrooms 

- planned developments 

= expected appreciation 
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REVISED CONSENSUS TREE 


BIG ENOUGH 


0.675 
FUNCTIONAL 


0.276 
LOCATION 
0.426 


MAINTAINABILITY 


suv? | suport 9.700 


1.000 0.376 
OURABILITY 


0.300 


iMM. AFFORD 
0.376 


INVESTMENT] CONT. AFFORO 
0.360 0.375 


FUTURE WORTH 
0.250 


OLD TREE 


360478 39 (6-27-83) 


SEDROOMS 
SQ. FOOTAGE 


0475 


0826 
SCHOOLS FUNCTIONAL 
0.463 0.233 


0.276 


work 


SHOPPING 
0.262 
YARO 


. 0.376 
AGE OF HOME 


0.626 Buy? | SUPPORT 
CONDITION 7.000 0.333 


0.625 
DISASTER vut. 

0.326 
OOWN 


CLOSING 


0.500 


0.600 


BASIC MONTHLY 
0.388 


INTEREST VAR. 
0400 INVESTMENT 

POSS. ASSESSMENT 0.434 
0.212 

EXP APPRECIATION 
0.425 


PLANNED DEv. 
0.875 


BRANCH VALUES RELATIVE TO 
EACH NODE 


REVISED CONSENSUS TREE 


BIG ENOUGH 
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FUNCTIONAL 
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LOCATION 
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MAINTAINABILITY 


BUY? SUPPORT o.262 


1.000 0.375 
DURABILITY 
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INVESTMENT] CONT. AFFORO 
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FUTURE WORTH 
0.088 


OLD TREE 
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BEDROOMS 
$Q. FOOTAGE 
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SCHOOLS 
0.054 0.233 


WORK 
0.032 
SHOPPING 
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YARO 
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AGE OF HOME 
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0.037 
DOWN 


CLOSING 


0.0666 


0.0656 
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BASIC MONTHL 0.061 


INTEREST VAR. INVESTMENT 
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0.028 


EXP. APPRECIATION 
0.037 


PLANNED O€EV. 
0.061 


BRANCH VALUES RELATIVE TO 
EACH LEVEL 
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BIG ENOUGH 
0.483 


LOCATION 
0.617 


MAINTAINABILITY 


0.700 


OURABILITY 
0.300 


IMM AFFORD 
0.317 


CONT AFFORD 
0.317 


FUTURE WORTH 
0.366 


NEW TREE 


BIG ENOUGH 
0.113 


LOCATION 
0.320 


MAINTAINABILITY 
0.233 


OURABILITY 
0.100 


IMM. AFFORD 
0.138 


CONT. AFFORD 
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FUTURE WORTH 
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NEW TREE 


BEDROOMS 
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SCHOOLS 
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SHOPPING 


YAAD 
AGE OF HOME 
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DISASTER VUE 


DOWN 
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Having completed the revised consensus tree, John can now procede to the 
assessment of the potential collection concepts. Since his parents had no 
disagreement with the information performance factors that were developed in 
the original analysis, the revised performance analysis follows much as 
before: relative to each collection concept, the revised relative values of 
information for each information requirement are multiplied by the expected 
performance of the concept and the resulting expected information values are 
summed to determine the overall expected information value collected. The 
results of this analysis are shown in the top figure of the next page. Once 
again it appears talking with the city or checking the newspaper fall short; 
this time, however, the margin between talking with his real estate friend and 
talking with the owner is greater. So the preliminary assessment is the third 
collection concept appears to be the most effective. 

To investigate this preference more thoroughly, John again considered 
the impact on the overall value of the information collected relative to each 
collection concept by evaluating each information requirement at its mininum 
and maximum levels of performance. The results, shown in the figure at the 
bottom of the next page, indicate that talking with the real estate friend 
appears to stochastically dominate all the other candidate collection 
concepts, however the issue of timeliness remains to be accounted for. 
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INFORMATION SOURCE EXP. VALUE 
TALK WITH THE OWNER | oss | 
TALK WITH THE CITY 


360479.41 (627 63) 


REVISED PERFORMANCE 
ANALYSIS 


TALK WITH 
TALK WITH FRIEND CHECK 
CITY (AGENT} NEWSPAPER . 


NUMBER OF BEDROOMS 
SQUARE FOOTAGE 

SCHOOLS (LOCATION) 

WORK (LOCATION) 

SHOPPING (LOCATION) 
AMOUNT OF YARD WORK 

AGE OF HOME 

CONDITION OF SIMILAR HOMES 
DISASTER VULNERABILITY 
DOWN PAYMENT 

CLOSING Costs 

BASIC MONTHLY PAYMENT 
PROJECTED INTEREST VARIANCE 
POSSIBLE ASSESSMENTS 
EXPECTED APPRECIATION 
PLANNED DEVELOPMENTS 
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As in the original analysis, John must adjust the assessment of the 
third collection concept, talking with the real estate friend, to reflect the 
two week delay in collecting any information. The same estimate is used for 
the discount rate, 5%, which is a factor of (1/1.05) or approximately 0.95. 
This translates into a reduction of about 3 to 4% in the performance factors 
for this collection concept as shown in the figure at the top of the next page. 

A review of the ranges for the performance factors associated with each 
of the collection concepts, adjusted for timeliness considerations (see bottom 
figure on next page), indicates that while the concepts of talking with the 
city and checking in the newspaper are indeed stochastically dominated, there 
is a slight overlap for the other two. So before John completes his analysis, 
he will do a probabilistic analysis on these two remaining concepts. 
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REVISED PERFORMANCE 
ADJUSTMENTS 


NUMBER OF BEDROOMS 
SQUARE FOOTAGE 

| SCHOOLS (LOCATION) 
WORK (LOCATION) 
SHOPPING (LOCATION) 
AMOUNT OF YARD WORK 
AGE OF HOME 
CONDITION OF SIMILAR HOMES 
DISASTER VULNERABILITY 
DOWN PAYMENT 
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EXPECTED APPRECIATION 
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The revised results of the probabilistic analysis are shown in the 
figures on the adjoining page. This time there is no question of which 
concept has the most effective performance. As the top figure shows, talking 
with the real estate friend consistently yields a likelihood for collecting a 
higher overall value of information. It is important to note, as the lower 
figure shows, this is completely independent of the range of performance of 
the two concepts, which was not changed; the third concept still has a greater 
possible variance in its performance. Because the relative weights of the 
information collected have changed, however, the overall expected performance 
of the two concepts has diverged, leaving the third concept the clear choice. 
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REVISED PERFORMANCE COMPARISON 


PROBABILITY 
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0.45 0.50 0.55 0.60 
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CONTINUING APPLICABILITY 


SIGNIFICANT AMOUNTS OF DATA ARE COMPILED TO 
SUPPORT THE METHODOLOGY, INCLUDING 


e KEY PROBLEM ISSUES 
@ CONCEPT PERFORMANCE ESTIMATES 


@ TIMELINESS AND SURVIVABILITY ADJUSTMENT 
~ FACTORS 


LARGE PORTIONS OF THAT DATA MAY BE REUSABLE 


e RELATED PROBLEMS MAY USE SOME — EITHER 
DIRECTLY OR AS A GUIDELINE 


@ FOLLOW-ON WORK FOR PROBLEM UNDER STUDY MAY 
USE SUBSTANTIAL PORTIONS 


As is readily apparent, the application of this methodology to 
collection systems problems is not a trivial exercise. Typical of all 
decision analysis approaches, it requires the compilation of a significant 
amount of data, including 

- key issues to be investigated, including the information required to 

support the process and each piece's relative weight 

- evaluation of experts 

= collection concept performance -- not just at the likely dr expected 
level but also minimum and maximum levels and, in some cases, an 
approximation of the entire performance distribution 

= quantification of impacts of discounting effects due to timeliness 

considerations and compounding effects due to survivability 
considerations 
As such, this approach should be reserved for large, resource-critical 
decisions, 

However, the usefulness of much of this data does not end with the 
study. Related problems may be able to use portions of the performance 
estimates or large subsections of the tree either intact or as a template or 
prototype. Similarly, there may be follow-on applications that reconsider 
some of the basic elements and leave the rest essentially the same. Examples 
might be the evaluation of an additional concept or the availability of a new 
expert or the analysis of the same problem but from a different point in time. 
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; Instead of initiating an information collection effort at the time the 
original study was done, John just did some preliminary investigation and 
delayed collecting everything else. A month later he decided to go ahead and 
seriously analyze the house in question. But the evaluation he would have 
done before was not completely correct any longer as he had already collected 
a certain amount of information (such as the number of bedrooms, the square 
footage, the relative distances, etc.). Therefore, collecting it again had 
significantly reduced importance. So he went back to his experts for updated 
weights. New consensus weights were calculated using the same evaluation of 
the experts as was done for the revised weights 

- 3°for Sue's value 

- 2 for Mary's value 

- 1 for Gary's value 
The updated weights supplied by the experts as well as the new consensus 
weights are shown in the top figure on the next page. These weights produce a 
new ordering of information, Again, a signficant re-ranking has taken place, 
especially for 

= age of house 

- amount of yard work 

- square footage 

- number of bedrooms 

- planned developments 

= expected appreciation 

= disaster vulnerability 

- possible assessments 
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UPDATED CALCULATION OF 
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UPDATED COMPARISON OF DATA 
PRIORITIES 


INFORMATION REQUIREMENT ORIGINAL REVISED UPDATED 


AGE OF HOUSE 1 
AMOUNT OF YARD WORK 3 
SQUARE FOOTAGE 8 
CONDITION OF SIMILAR HOUSES 7 

5 

6 

7 

8 
9 7 
10 8 
11 1 
12 5 


s 


HUGHES 


. 


NUMBER OF BEDROOMS 
DOWN PAYMENT 
CLOSING COSTS 
LOCATION RELATIVE TO SCHOOLS 


INTEREST VARIANCE 
BASIC MONTHLY PAYMENT 
PLANNED DEVELOPMENTS 
DISASTER VULNERABILITY 


EXPECTED APPRECIATION 
LOCATION RELATIVE TO WORK 

LOCATION RELATIVE TO SHOPPING 
POSSIBLE ASSESSMENTS 
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With the updated consensus tree, John procedes to the collection concept 
evaluation phase. Assuming the same set of collection concepts with the same 
performance characteristics, an analysis is done of the updated performance, 
generating overall expected values for the information collected for each 
concept. The results are presented at the top of the next page. This time 
talking with the friend appears to be significantly more effective than any 
other concept. 

To verify that talking with the friend is the concept to be chosen, John 
considered for a third time the impact on overall value of information 
collected relative to each collection concept from evaluating each information 
requirement ‘at its minimum and maximum levels of peformance. The results, 
shown in the figure at the bottom of the next page, demonstrate unequivically 
that the third concept stochastically dominates the entire set. Therefore, 
there is no need for further analysis. 
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UPDATED EXPECTED PERFORMANCE 


INFORMATION SOURCE EXP. VALUE 


TALK WITH THE OWNER [we 
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IV. DEMONSTRATION EFFORT 


DEMONSTRATION EFFORT 


PROBLEM SELECTION CRITERIA 


_@ CONSENSUS PROBLEM 


e IMPORTANCE OF PROBLEM MUTUALLY AGREED 
UPON 


e MUST DEMONSTRATE ADAPTABILITY OF 
METHODOLOGY 


e POSSESS A DEGREE OF COMPLEXITY TO EXERCISE 
ALL FACETS OF METHODOLOGY 


e MUST ALLOW FOR APPLICATION OF SENSITIVITY 
ANALYSIS 


200479-32 (627-43) 


For the methodology developed to be of use to the analyst, it must be 
easily applied to the problems with which he/she is concerned. The preceding 
"toy" example is useful for illustration and education, but a demonstration 
effort on a more relevant problem to the collection concept analyst is 
needed. Care should be taken in choosing the topic of this effort so that 
maximum benefit may be achieved. 

First of all, the problem should be of enough importance to warrant the 
effort undertaken and be of relevance to the community. The problem should 
have sufficient complexity to fully utilize and illustrate the documented 
methodology. Finally, the problem should be such that sensitivity analysis 
can be meaningfully employed and be used to show the conclusions and results 
such an effort makes possible. 

In summary, the demonstration effort should show the power and 
usefulness of the methodology as well as its flexibility and adaptability. 
The remainder of this report documents what has been completed to date 
relative to this effort. 


ee 
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IV. A. SAM PROBLEM 


prt ttre cence cece 


SAM PROBLEM : HUGHES : 


eee ere ee) 


WHAT EFFECT WILL THE INTRODUCTION OF 
A SURFACE TO AIR MISSILE HAVE ON 
THE BALANCE OF POWER? 


7806 79-47 (6-27-63) 


The final stage of this analysis involves applying the developed 
methodology to a problem of interest to the intelligence community. By 
documenting this process, an example is available to thé analyst to use‘as a 
guideline when performing additional analyses relating to collection concépt 
concerns, 

The problem chosen for illustration involves the information required to 
answer the question: What effect will the introduction of a surface to air 
missile (SAM) have on the balance of power? The point to keep in mind here is 
the frame of reference for the analysis. The methodology does not attempt to 
answer the question regarding the SAM and its effects, but instead the § ~ 
methodology determines the information required to answer the question. Given 
these requirements, the ability of a particular collection concept to collect 
the desired information can be evaluated and the rest of ‘the sensitivity 
analysis completed. 

As of the writing of this interim report, the SAM problem has been 
completed through the structuring of the information tree the structure is 
{llustrated on the following pages. The remainder of the analysis will be 
documented in the final report of this study. 
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IV. B. INFORMATION TREE STRUCTURE 
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VULNERABILITY NON-RADIATION HOMING MISSIL EKVI) 


SOFT COUNTERS ecm Svip 
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The top level tree for the SAM problem is shown above. The level of 
information is seen to get more specific with descent through the tree. For 
example, the first division is between direct threats posed by the system and 
modifications that could be undertaken to regain the balance of power. The 
tree expands in detail until the raw or collectable data level is reached. 
The tree structure in its entirety is illustrated on the following pages and 
is indexed using Roman numerals. The numerals refer to the figure number on 
which the continuation of the tree can be found. For example, the expansion 
of the homeland deployment tree can be found on Figure VIII. 
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e SENSITIVITY ANALYSIS 


The major product of this interim report is the methodology which has 
been developed to perform collection concept analyses. The methodology has 
been shown to be adaptable to a wide range of problems, is easily used to 
track the analysis path, and is able to make use of many types of data. The 
method handles both quantitative and quantitative data as well as allowing for 
several experts opinions to be aggregated to form a consensus opinion. Most 
importantly, the method allows for sensitivity analysis which provides for 
critical review between choices as well as determination of confidence levels 
in the conclusions reached. 
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